Dataflow controller technology for dataflow execution graph

ABSTRACT

A dataflow controller technology. A dataflow execution graph (such as a directed acyclic graph) receives or is to receive one or more data streams for processing. The control of the dataflow execution graph is performed by a set of one or more dataflow controllers that are dedicated to that dataflow execution graph. Each dataflow execution graph is instantiated and assigned to the dataflow execution graph such that the assignment results in the dataflow controller executing dataflow control functions for that dataflow execution graph. By dedicating controller(s) to only one dataflow execution graph, the risk of failure of the controller is bound to a single dataflow execution graph.

BACKGROUND

Large scale cloud and Internet service providers typically generatemillions of events per second. To handle such high event throughput,events are often accumulated, prior to being processed as a batch. Morerecently, to reduce latency and to ensure timely event processing,stream processing systems avoid batching by processing the events as astream.

There can be high variability (called herein “temporal variability”) inthe volume of events that are being streamed with each event stream. Forinstance, an event stream can include a mix of expected events (e.g.,processing needs during the day can be typically higher than at night,and so forth), and unexpected events (e.g., dramatic stock marketchanges, and so forth). Furthermore, each event stream has differentresource requirements due to there being different workloadcharacteristics (called herein “spatial variability”) across eventstreams. Furthermore, in large-scale systems, there are inevitablefailures and hardware heterogeneity that make it hard to ensure stableperformance in processing event streams. To handle these variabilitiesand uncertainties, users of stream processing systems (typically systemadministrators) often provision resources with a safety factor, leavingmany resources idle or underutilized.

Many existing stream processing systems adopt a streaming dataflowcomputational model. In this model, a computational job is representedas a directed acyclic graph (DAG) of operators, which is also called a“dataflow execution graph”. Although such operators may be stateless,such operators are most often stateful in that they maintain mutablelocal state. Each operator sends and/or receives logically timestampedevents along directed edges of the DAG. Upon receiving an event along aninput edge(s), an operator updates its local state if appropriate,potentially generates new events, and sends those new events todownstream operators along output edge(s). Operators without input edgesare termed “source” operators, or simply “sources”. Operators withoutoutput edges are termed “sink” operators, or simply “sinks”. An edge ina DAG has no state but can have configurable properties. For example, aproperty of an edge might be queue size thresholds that triggerback-pressure.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments such asthose described above. Rather, this background is only provided toillustrate one exemplary technology area where some embodimentsdescribed herein may be practiced.

BRIEF SUMMARY

At least some embodiments described herein relate to a dataflowcontroller technology. A dataflow execution graph (such as a directedacyclic graph) receives or is to receive one or more data streams forprocessing. The control of the dataflow execution graph is performed bya set of one or more dataflow controllers that are dedicated to thatdataflow execution graph. Each dataflow execution graph is instantiatedand assigned to the dataflow execution graph such that the assignmentresults in the dataflow controller executing dataflow control functionsfor that dataflow execution graph.

By dedicating controller(s) to only one dataflow execution graph, therisk of failure of the controller is bound to a single dataflowexecution graph. Thus, if a dataflow controller fails, there is only onedataflow execution graph impacted. The concept may be implemented in anenvironment in which there are multiple dataflow execution graphs, inwhich there may be one set of dataflow controller(s) for one dataflowexecution graph, another set of dataflow controller(s) for anotherdataflow execution graph, and so forth.

There may be a higher-level controller (or a “multi-dataflowcontroller”) for different sets of dataflow controller(s). This type ofcontrol allows execution to be much more resilient to failure of acontroller. For instance, a higher-level controller may perform controltasks for multiple dataflow controllers. However, functions that areallocated to the higher-level controller may be less subject to risk offailure (like simply periodically making sure a dataflow controller isrunning), whereas the dataflow controllers may perform moreprocessing-intensive functions that are more subject to risk of failure,such as migrating a dataflow execution graph from one structure toanother in order to perform scale out. Alternatively, or in addition,the higher-level controllers may be redundantly provisioned.

As another example, dataflow controllers may be redundantly assigned toa given dataflow controller. Alternatively, or in addition, a pool ofpre-instantiated controllers may be available to be assigned when neededto a particular dataflow execution graph. Should a dataflow controllerfail, the pre-instantiated controller may be quickly assigned to thedataflow execution graph. By allocating controllers in this way, failurerisk may be more easily managed resulting in higher reliability ofcontrol.

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the invention can be obtained, a moreparticular description of the invention briefly described above will berendered by reference to specific embodiments thereof which areillustrated in the appended drawings. Therefore, these drawings depictonly example embodiments of the invention and are not therefore to beconsidered to be limiting of the scope of the invention. With this inmind, example embodiments of the invention will be described andexplained with reference to the accompanying drawings in which:

FIG. 1 illustrates an example dataflow execution graph in the form of adirected acyclic graph (DAG) that has multiple operators and edges, andis provided as merely one example of an innumerable variety of dataflowexecution graphs;

FIG. 2 illustrates an environment in which the principles describedherein may be employed, which includes multiple dataflow executiongraphs, a dataflow controller set for each of the illustrated dataflowexecution graphs, and a multi-dataflow controller set that providecontrollers that control across dataflow execution graphs;

FIG. 3 illustrates a more specific example of the environment of FIG. 2,in which there are specific example dataflow execution graphs shown,particular dataflow controllers shown (including role-based dataflowcontrollers), and higher-level multi-dataflow controllers illustrated inhierarchical form;

FIG. 4 illustrates a pool of pre-instantiated controllers that may beused to supplement the controllers of FIGS. 2 and 3;

FIG. 5 illustrates a flowchart of a method for configuring control of adataflow execution graph in accordance with the principles describedherein; and

FIG. 6 illustrates an example computer system in which the principlesdescribed herein may be employed.

DETAILED DESCRIPTION

At least some embodiments described herein relate to a dataflowcontroller technology. A dataflow execution graph (such as a directedacyclic graph) receives or is to receive one or more data streams forprocessing. The control of the dataflow execution graph is performed bya set of one or more dataflow controllers that are dedicated to thatdataflow execution graph. Each dataflow execution graph is instantiatedand assigned to the dataflow execution graph such that the assignmentresults in the dataflow controller executing dataflow control functionsfor that dataflow execution graph. By dedicating controller(s) to onlyone dataflow execution graph, the risk of failure of the controller isbound to a single dataflow execution graph. Thus, if a dataflowcontroller fails, there is only one dataflow execution graph impacted.

First, the concept of a dataflow execution graph will be described withrespect to FIG. 1. Then, an example environment in which multiplecontrollers control multiple dataflow execution graphs will be describedwith respect to FIGS. 2 and 3. The use of a pre-instantiated controllerpool that allows for rapid assignment of controllers withoutenvironments of FIGS. 2 and 3 will then be described with respect toFIG. 4. Then, a method for configuring control of a dataflow executiongraph will be described with respect to the flowchart illustrated inFIG. 5. Finally, because various aspects described herein may beimplemented by an executable component of a computing system, acomputing system will be described with respect to FIG. 6.

In the streaming dataflow computational model, a computational job isrepresented as a directed acyclic graph (DAG) of operators, which isalso called a dataflow execution graph. Although such operators may bestateless, such operators are often stateful in that they maintainmutable local state. Each operator sends and/or receives logicallytimestamped data messages along directed edges of the dataflow executiongraph. Upon receiving a data message along an input edge, an operatorupdates its local state if appropriate, potentially generates new datamessages, and sends those new data messages to downstream operatorsalong output edge(s).

Operators without input edges are termed “source” operators, or simply“sources”. These source operators receive the raw input data messages ofthe data stream(s). For instance, one or more of the source operatorsmay receive data messages from one data stream, and one or more other ofthe source operators may receive data messages from another data stream,and so forth. Operators without output edges are termed “sink”operators, or simply “sinks”. One or more of these sink operatorsgenerate the output result of the dataflow execution graph.

A dataflow execution graph may include any number of operators and anynumber of edges in any configuration. The dataflow execution graph maybe as simple as a single operator, with zero edges. On the other hand,the dataflow execution graph may be indescribably complex, havinginnumerable operators and edges therebetween. FIG. 1 illustrates arelatively simple example in the form of a dataflow execution graph 100.

The dataflow execution graph 100 includes four operators 101, 102, 103and 104, and four directed edges 111, 112, 113 and 114. The directededges generally are directed rightward. The operators 101 and 102 aresource operators, each receiving the input data messages of the datastreams. As an example, perhaps source operator 101 receives input datamessages from one data stream (as represented by arrow 121), and sourceoperator 102 receives input data messages from another data stream (asrepresented by arrow 122). The operator 101 performs an operation on thedata messages of the input data stream 121 and provides resulting datamessages along directed edges 111 and 112 to appropriate respectiveoperators 103 and 104. The operator 102 performs an operation on thedata messages of another input data stream 122 and provides resultingdata messages along directed edges 113 and 114 to appropriate respectiveoperators 103 and 104. The operators 103 and 104 are sink operators,each providing output (as represented by the arrows 123 and 124) fromthe dataflow execution graph 100.

Now that dataflow execution graphs have been described, an exampleenvironment in which multiple controllers control multiple dataflowexecution graphs will be described with respect to FIG. 2. FIG. 2illustrates an environment 200 in which the principles described hereinmay be employed. The environment 200 includes any number of dataflowexecution graphs 220. In the illustrated embodiment, there are threeillustrated dataflow execution graphs 221, 222 and 223. However, theellipsis 224 represents that the principles described herein are notlimited to the number of dataflow execution graphs 220.

The environment 200 also includes dataflow controller sets 210. Forinstance, in the illustrated case, each of the illustrated dataflowexecution graphs has an associated dataflow controller set. In thisdescription, a “dataflow controller” is a controller that is assigned toa particular dataflow execution graph so as to execute control functionsfor that specific dataflow execution graph. If that dataflow controllerwas to fail, only control functions for that specific dataflow executiongraph would be impacted.

For instance, the first dataflow execution graph 221 has the associateddataflow controller set 211, the second dataflow execution graph 222 hasan associated dataflow controller set 212, and the third dataflowexecution graph 223 has an associated dataflow controller set 213.However, the ellipsis 214 represents that the principles describedherein are not limited to the number of dataflow controller sets 210.The ellipsis 214 also represents that the principles described herein donot require that all of the dataflow execution graphs in the environment200 have an associated dataflow controller set. Perhaps only some of thedataflow execution graphs 220 in the environment 200 have an associateddataflow controller set. Each dataflow controller set includes one ormore dataflow controllers.

The environment 200 also may potentially include a controller set 201that is not specific to any particular dataflow execution graph. Forinstance, the controller set 201 may include controllers that controlmultiple of the dataflow controllers. Such a controller will also bereferred to herein as a “multi-dataflow” controller. Each controller ofthe dataflow controller sets 220 or the controller set 201 may be anexecutable component (such as the executable component 606 of FIG. 6described further below) that is executed by a computing system (such asthe computing system 600 of FIG. 6 also described further below).

FIG. 3 illustrates an environment 300 that represents an example of theenvironment 200 of FIG. 2. Here, the dataflow execution graphs 320 ofthe environment 300 of FIG. 3 are examples of the dataflow executiongraphs 220 of the environment 200 of FIG. 2. The first dataflowexecution graph 321 of FIG. 3 is an example of the first dataflowexecution graph 221 of FIG. 2, and is similar to the dataflow executiongraph 100 of FIG. 1. The second dataflow execution graph 322 of FIG. 3is an example of the second dataflow execution graph 222 of FIG. 2, andis different than the dataflow execution graph 100 of FIG. 1. The thirddataflow execution graph 323 of FIG. 3 is an example of the thirddataflow execution graph 223 of FIG. 2, and is different than the othertwo dataflow execution graphs 321 and 322.

In FIG. 3, the dataflow execution graph 321 has an associated dataflowcontroller set 311, which is an example of the dataflow controller set211 of FIG. 2. The dataflow controller set 311 has four dataflowcontrollers 311AA, 311AB, 311B and 311C. The dataflow controller set 311contains role-based dataflow controllers, where each role is symbolizedby the respective controller being represented as a particular shape.For instance, dataflow controllers 311AA and 311AB are triangular whichsymbolizes that they perform a first role, dataflow controller 311B issquare which symbolizes that it performs a second role that is differentthan the first role, and dataflow controller 301C is a pentagon whichsymbolizes that it performs a third role that is different than thefirst and second roles. Examples of different roles that may beperformed by a role-based dataflow controller may be 1) monitoring thehealth of execution of the dataflow execution graph, 2) checkpointing astate of the dataflow execution graph, 3) reconfiguring the dataflowexecution graph, and so forth. The reconfiguring may comprise changingan actual structure of the dataflow execution graph.

Note that there are two instances 311AA and 311AB of the dataflowcontroller that perform the first role within the first dataflowcontroller set 311. Thus, the dataflow controllers may have redundantcopies running at the same time to ensure that if one of the role-baseddataflow controllers 311AA or 311AB fails, there is another role-baseddataflow controller 311AA or 311AB that can continue performing therole. This may be beneficial where the role is particularly critical orsensitive to downtime.

The dataflow controller set 312 has three dataflow controllers 312A,312B and 312C. The dataflow controllers of the dataflow controller set312 are also role-based controllers with dataflow controller 312Aperforming the first role (as symbolized by its shape being a triangle),dataflow controller 312B performing the second role (as symbolized byits shape being a square), and dataflow controller 312C performing thethird role (as symbolized by its shape being a pentagon).

The dataflow controller set 313 has a single dataflow controller 313A.This dataflow controller 313A is not role-based, but performs morecomprehensive control of its corresponding dataflow execution graph 323.Thus, FIG. 3 is provided as an example to illustrate the principle thatthe dataflow controller set for any given dataflow execution graph maybe as few as a single dataflow controller, or may include many dataflowcontrollers. Furthermore, FIG. 3 is provided to demonstrate that thedataflow controllers may perform certain roles only, or may be moregeneralized controllers. Also, FIG. 3 is provided to show that thedataflow controllers may be redundantly provided so as to continueproper control if a dataflow controller fails.

In FIG. 3, the environment 300 also includes a multi-dataflow controllerset 301, which is an example of the controller set 201 of FIG. 2. Themulti-dataflow controller set 301 is illustrated as including fourmulti-dataflow controllers 301A, 301B, 301C and 301D. However, theellipses 302 represent that the multi-dataflow controller set 301 mayinclude any number of controllers. The multi-dataflow controller set 301includes three multi-dataflow controllers 301A, 301B and 301C that arerole-based and supervise control of dataflow controllers of a particularrole. The multi-dataflow controller set 301 also includes amulti-dataflow controller 301D that performs more supervisory functionsof a general nature. The ellipsis 302 also represents that themulti-dataflow controller set 301 is not limited to the number (if anyat all) of role-based or general controllers, nor to how suchcontrollers are hierarchically structured, if hierarchically structuredat all.

In the illustrated example of FIG. 3, the multi-dataflow controller set301 includes a role-based multi-dataflow controller 301A that controlsoperation of the dataflow controllers 311AA, 311AB and 312A that performthe first role. The multi-dataflow controller set 301 also includes arole-based multi-dataflow controller 301B that controls operation of thedataflow controllers 311B and 312B that perform the second role, and arole-based multi-dataflow controller 301C that controls operation of thedataflow controllers 311C and 312C that perform the third role.

The multi-dataflow controller set 301 also includes a controller 301Dthat controls operation of several of the other multi-dataflowcontrollers 301A, 301B, and 301C, as well as one of the dataflowcontrollers 313A. Thus, although FIG. 3 is provided by way of exampleonly, FIG. 3 is used to show that there may be one or moremulti-dataflow controllers that control the dataflow controllers, andthat the multi-dataflow controllers may be hierarchically structuredsuch that a multi-dataflow controller may control some of the dataflowcontrollers via one or more intermediate-level controllers.

The ellipses 302 also symbolizes that any of the multi-dataflowcontrollers may have a redundant copy of itself so as to prevent failureof control should that controller fail.

The principles described herein allow risk of controller failure to becarefully managed, thereby providing a control plane that is robust tofailure. By dedicating dataflow controller(s) to only one dataflowexecution graph, the risk of failure of the dataflow controller is boundto a single dataflow execution graph. Thus, if a dataflow controllerfails, there is only one dataflow execution graph impacted. However,robustness against failure may be accomplished in other ways also usingthe principles described herein.

For instance, functions that are allocated to the higher-levelcontroller may be less subject to risk of failure (like simplymonitoring the health of the dataflow controllers or even just makingsure the dataflow controllers are still running), whereas the dataflowcontrollers may perform more processing-intensive functions that aremore subject to risk of failure, such as migrating a dataflow executiongraph from one structure to another in order to perform scale out.Alternatively, or in addition, the higher-level controllers may beredundantly provided. As another example, dataflow controllers may beredundantly assigned to a given dataflow controller.

One additional example for protection against failure is illustrated inFIG. 4, which shows a pool of pre-instantiated controllers that may beassigned when needed to satisfy a particular controller deficiency inthe environments 200 and 300. For instance, FIG. 4 illustrates acontroller pool 400 that includes spare role-based dataflow controllers411A, 411B and 411C, spare general dataflow controller 411D, sparerole-based multi-dataflow controllers 401A, 401B and 401C, and a sparegeneral multi-dataflow controller 401D. By having these controllerspre-instantiated, they need not be instantiated prior to being importedinto the working environment. Thus, downtime and loss of redundancy dueto a controller failure may be more quickly addressed.

FIG. 5 illustrates a flowchart of a method for configuring control of adataflow execution graph in accordance with the principles describedherein. The dataflow controller is first instantiated (act 501). Thisinstantiation may occur well in advance, as when populating thecontroller pool 400 of FIG. 4. Alternatively, new controllers may beinstantiated only after a deficiency in controllers is discovered withinthe environment 200 or 300.

Then, since the controller is a dataflow controller, that dataflowcontroller is assigned to a dataflow execution graph (act 502). Forinstance, in FIG. 2, any of the dataflow controllers in the dataflowcontroller set 211 would be assigned to the dataflow execution graph221, any of the dataflow controllers in the dataflow controller set 212would be assigned to the dataflow execution graph 222, and any of thedataflow controllers in the dataflow controller set 213 would beassigned to the dataflow execution graph 223. Likewise, in FIG. 3, anyof the dataflow controllers in the dataflow controller set 311 would beassigned to the dataflow execution graph 321, any of the dataflowcontrollers in the dataflow controller set 312 would be assigned to thedataflow execution graph 322, and any of the dataflow controllers in thedataflow controller set 313 would be assigned to the dataflow executiongraph 323. The method 500 of FIG. 5 may be performed multiple times foreach dataflow execution graph, and for each of multiple dataflows.

Because the principles described herein operate in the context of acomputing system, a computing system will be described with respect toFIG. 6. Computing systems are now increasingly taking a wide variety offorms. Computing systems may, for example, be handheld devices,appliances, laptop computers, desktop computers, mainframes, distributedcomputing systems, datacenters, or even devices that have notconventionally been considered a computing system, such as wearables(e.g., glasses, watches, bands, and so forth). In this description andin the claims, the term “computing system” is defined broadly asincluding any device or system (or combination thereof) that includes atleast one physical and tangible processor, and a physical and tangiblememory capable of having thereon computer-executable instructions thatmay be executed by a processor. The memory may take any form and maydepend on the nature and form of the computing system. A computingsystem may be distributed over a network environment and may includemultiple constituent computing systems.

As illustrated in FIG. 6, in its most basic configuration, a computingsystem 600 typically includes at least one hardware processing unit 602and memory 604. The memory 604 may be physical system memory, which maybe volatile, non-volatile, or some combination of the two. The term“memory” may also be used herein to refer to non-volatile mass storagesuch as physical storage media. If the computing system is distributed,the processing, memory and/or storage capability may be distributed aswell.

The computing system 600 has thereon multiple structures often referredto as an “executable component”. For instance, the memory 604 of thecomputing system 600 is illustrated as including executable component606. The term “executable component” is the name for a structure that iswell understood to one of ordinary skill in the art in the field ofcomputing as being a structure that can be software, hardware, or acombination thereof. For instance, when implemented in software, one ofordinary skill in the art would understand that the structure of anexecutable component may include software objects, routines, methodsthat may be executed on the computing system, whether such an executablecomponent exists in the heap of a computing system, or whether theexecutable component exists on computer-readable storage media.

In such a case, one of ordinary skill in the art will recognize that thestructure of the executable component exists on a computer-readablemedium such that, when interpreted by one or more processors of acomputing system (e.g., by a processor thread), the computing system iscaused to perform a function. Such structure may be computer-readabledirectly by the processors (as is the case if the executable componentwere binary). Alternatively, the structure may be structured to beinterpretable and/or compiled (whether in a single stage or in multiplestages) so as to generate such binary that is directly interpretable bythe processors. Such an understanding of example structures of anexecutable component is well within the understanding of one of ordinaryskill in the art of computing when using the term “executablecomponent”.

The term “executable component” is also well understood by one ofordinary skill as including structures that are implemented exclusivelyor near-exclusively in hardware, such as within a field programmablegate array (FPGA), an application specific integrated circuit (ASIC), orany other specialized circuit. Accordingly, the term “executablecomponent” is a term for a structure that is well understood by those ofordinary skill in the art of computing, whether implemented in software,hardware, or a combination. In this description, the term “component” or“vertex” may also be used. As used in this description and in the case,this term (regardless of whether the term is modified with one or moremodifiers) is also intended to be synonymous with the term “executablecomponent” or be specific types of such an “executable component”, andthus also have a structure that is well understood by those of ordinaryskill in the art of computing.

In the description that follows, embodiments are described withreference to acts that are performed by one or more computing systems.If such acts are implemented in software, one or more processors (of theassociated computing system that performs the act) direct the operationof the computing system in response to having executedcomputer-executable instructions that constitute an executablecomponent. For example, such computer-executable instructions may beembodied on one or more computer-readable media that form a computerprogram product. An example of such an operation involves themanipulation of data.

The computer-executable instructions (and the manipulated data) may bestored in the memory 604 of the computing system 600. Computing system600 may also contain communication channels 608 that allow the computingsystem 600 to communicate with other computing systems over, forexample, network 610.

While not all computing systems require a user interface, in someembodiments, the computing system 600 includes a user interface 612 foruse in interfacing with a user. The user interface 612 may includeoutput mechanisms 612A as well as input mechanisms 612B. The principlesdescribed herein are not limited to the precise output mechanisms 612Aor input mechanisms 612B as such will depend on the nature of thedevice. However, output mechanisms 612A might include, for instance,speakers, displays, tactile output, holograms, virtual reality, and soforth. Examples of input mechanisms 612B might include, for instance,microphones, touchscreens, holograms, virtual reality, cameras,keyboards, mouse of other pointer input, sensors of any type, and soforth.

Embodiments described herein may comprise or utilize a special purposeor general-purpose computing system including computer hardware, suchas, for example, one or more processors and system memory, as discussedin greater detail below. Embodiments described herein also includephysical and other computer-readable media for carrying or storingcomputer-executable instructions and/or data structures. Suchcomputer-readable media can be any available media that can be accessedby a general purpose or special purpose computing system.Computer-readable media that store computer-executable instructions arephysical storage media. Computer-readable media that carrycomputer-executable instructions are transmission media. Thus, by way ofexample, and not limitation, embodiments can comprise at least twodistinctly different kinds of computer-readable media: storage media andtransmission media.

Computer-readable storage media includes RAM, ROM, EEPROM, CD-ROM orother optical disk storage, magnetic disk storage or other magneticstorage devices, or any other physical and tangible storage medium whichcan be used to store desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computing system.

A “network” is defined as one or more data links that enable thetransport of electronic data between computing systems and/or componentsand/or other electronic devices. When information is transferred orprovided over a network or another communications connection (eitherhardwired, wireless, or a combination of hardwired or wireless) to acomputing system, the computing system properly views the connection asa transmission medium. Transmissions media can include a network and/ordata links which can be used to carry desired program code means in theform of computer-executable instructions or data structures and whichcan be accessed by a general purpose or special purpose computingsystem. Combinations of the above should also be included within thescope of computer-readable media.

Further, upon reaching various computing system components, program codemeans in the form of computer-executable instructions or data structurescan be transferred automatically from transmission media to storagemedia (or vice versa). For example, computer-executable instructions ordata structures received over a network or data link can be buffered inRAM within a network interface component (e.g., a “NIC”), and theneventually transferred to computing system RAM and/or to less volatilestorage media at a computing system. Thus, it should be understood thatreadable media can be included in computing system components that also(or even primarily) utilize transmission media.

Computer-executable instructions comprise, for example, instructions anddata which, when executed at a processor, cause a general-purposecomputing system, special purpose computing system, or special purposeprocessing device to perform a certain function or group of functions.Alternatively, or in addition, the computer-executable instructions mayconfigure the computing system to perform a certain function or group offunctions. The computer executable instructions may be, for example,binaries or even instructions that undergo some translation (such ascompilation) before direct execution by the processors, such asintermediate format instructions such as assembly language, or evensource code.

Those skilled in the art will appreciate that the invention may bepracticed in network computing environments with many types of computingsystem configurations, including, personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, pagers, routers, switches, datacenters, wearables (such as glassesor watches) and the like. The invention may also be practiced indistributed system environments where local and remote computingsystems, which are linked (either by hardwired data links, wireless datalinks, or by a combination of hardwired and wireless data links) througha network, both perform tasks. In a distributed system environment,program components may be located in both local and remote memorystorage devices.

Those skilled in the art will also appreciate that the invention may bepracticed in a cloud computing environment, which is supported by one ormore datacenters or portions thereof. Cloud computing environments maybe distributed, although this is not required. When distributed, cloudcomputing environments may be distributed internationally within anorganization and/or have components possessed across multipleorganizations.

In this description and the following claims, “cloud computing” isdefined as a model for enabling on-demand network access to a sharedpool of configurable computing resources (e.g., networks, servers,storage, applications, and services). The definition of “cloudcomputing” is not limited to any of the other numerous advantages thatcan be obtained from such a model when properly deployed.

For instance, cloud computing is currently employed in the marketplaceso as to offer ubiquitous and convenient on-demand access to the sharedpool of configurable computing resources. Furthermore, the shared poolof configurable computing resources can be rapidly provisioned viavirtualization and released with low management effort or serviceprovider interaction, and then scaled accordingly.

A cloud computing model can be composed of various characteristics suchas on-demand, self-service, broad network access, resource pooling,rapid elasticity, measured service, and so forth. A cloud computingmodel may also come in the form of various application service modelssuch as, for example, Software as a service (“SaaS”), Platform as aservice (“PaaS”), and Infrastructure as a service (“IaaS”). The cloudcomputing model may also be deployed using different deployment modelssuch as private cloud, community cloud, public cloud, hybrid cloud, andso forth. In this description and in the claims, a “cloud computingenvironment” is an environment in which cloud computing is employed.

Accordingly, the principles described herein allow risk of controllerfailure to be carefully managed, thereby providing a control plane thatis robust to failure. By dedicating dataflow controller(s) to only onedataflow execution graph, the risk of failure of the dataflow controlleris bound to a single dataflow execution graph. Thus, if a dataflowcontroller fails, there is only one dataflow execution graph impacted.Additional mechanisms for providing robustness against controllerfailure have also been described.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

1. A computing system comprising: one or more processors; and one ormore computer-readable storage media having thereon computer executableinstructions that are structured such that, when executed by the one ormore processors, the computing system is caused configure control of aplurality of dataflow execution graphs, based on performing at least thefollowing: instantiating a plurality of dataflow controllers, includinginstantiating at least one dataflow controller for each of the pluralityof dataflow execution graphs, instantiating each dataflow controllerincluding instantiating a corresponding plurality of role-based dataflowcontrollers that each performs a different type of operation on adataflow execution graph associated with the dataflow controller; andassigning each of the plurality of dataflow controllers to a differentcorresponding dataflow execution graph of the plurality of dataflowexecution graphs, each dataflow execution graph being configured toreceive one or more data streams for processing by the dataflowexecution graph, the assignment of each dataflow controller to acorresponding dataflow execution graph resulting in the plurality ofrole-based dataflow controllers corresponding to the dataflow controllerexecuting dataflow control functions for the corresponding dataflowexecution graph.
 2. (canceled)
 3. (canceled)
 4. (canceled)
 5. (canceled)6. (canceled)
 7. The computing system in accordance with claim 1,wherein, for at least one of the plurality of dataflow controllers, oneof the plurality of role-based dataflow controller performs a role ofmonitoring a health of execution of the dataflow execution graphassigned to the at least one of the plurality of dataflow controllers.8. The computing system in accordance with claim 1, wherein, for atleast one of the plurality of dataflow controllers, one of the pluralityof role-based dataflow controller performs a role of checkpointing astate of the dataflow execution graph assigned to the at least one ofthe plurality of dataflow controllers.
 9. The computing system inaccordance with claim 1, wherein, for at least one of the plurality ofdataflow controllers, one of the plurality of role-based dataflowcontroller performs a role of reconfiguring the dataflow execution graphassigned to the at least one of the plurality of dataflow controllers.10. The computing system in accordance with claim 9, the reconfiguringcomprising changing a structure of the dataflow execution graph.
 11. Thecomputing system in accordance with claim 1, the computer-executableinstructions also being structured such that, when executed by the oneor more processors, the computing system is caused to insatiate at leastone multi-dataflow controller that controls multiple of the plurality ofdataflow controllers.
 12. (canceled)
 13. The computing system inaccordance with claim 11, the multi-dataflow controller controlling atleast some of the multiple of the plurality of dataflow controllers viaat least one intermediate-level dataflow controller.
 14. A method forconfiguring control of a plurality of dataflow execution graphs, themethod comprising the following: instantiating a plurality of dataflowcontrollers, including instantiating at least one dataflow controllerfor each of the plurality of dataflow execution graphs, instantiatingeach dataflow controller including instantiating a correspondingplurality of role-based dataflow controllers that each performs adifferent type of operation on a dataflow execution graph associatedwith the dataflow controller; and assigning each of the plurality ofdataflow controllers to a different corresponding dataflow executiongraph of the plurality of dataflow execution graphs, each dataflowexecution graph being configured to receive one or more data streams forprocessing by the dataflow execution graph, the assignment of eachdataflow controller to a corresponding dataflow execution graphresulting in the plurality of role-based dataflow controllerscorresponding to the dataflow controller executing dataflow controlfunctions for the corresponding dataflow execution graph.
 15. (canceled)16. The method in accordance with claim 14, wherein, for at least one ofthe plurality of dataflow controllers, the plurality of role-baseddataflow controller include at least one of the following: a set ofrole-based dataflow controller(s) that performs a role of monitoring ahealth of execution of the dataflow execution graph assigned to the atleast one of the plurality of dataflow controllers; a set of role-baseddataflow controller(s) that performing a role of checkpointing a stateof the dataflow execution graph assigned to the at least one of theplurality of dataflow controllers; or a set of role-based dataflowcontroller(s) that perform a role of reconfiguring the dataflowexecution graph assigned to the at least one of the plurality ofdataflow controllers.
 17. The method of claim 14, further comprising:instantiating at least one multi-dataflow controller that controlsmultiple of the plurality of dataflow controllers.
 18. (canceled) 19.The method in accordance with claim 17, the multi-dataflow controllercontrolling at least some of the multiple of the plurality of dataflowcontrollers via at least one intermediate-level dataflow controller. 20.A computer program product comprising one or more computer-readablestorage media having thereon computer-executable instructions that arestructured such that, when executed by one or more processors of acomputing system, cause the computing system to configure control of aplurality of dataflow execution graphs, based on performing at least thefollowing: instantiating a plurality of dataflow controllers, includinginstantiating at least one dataflow controller for each of the pluralityof dataflow execution graphs, instantiating each dataflow controllerincluding instantiating a corresponding plurality of role-based dataflowcontrollers that each performs a different type of operation on adataflow execution graph associated with the dataflow controller; andassigning each of the plurality of dataflow controllers to a differentcorresponding dataflow execution graph of the plurality of dataflowexecution graphs, each dataflow execution graph being configured toreceive one or more data streams for processing by the dataflowexecution graph, the assignment of each dataflow controller to acorresponding dataflow execution graph resulting in the plurality ofrole-based dataflow controllers corresponding to the dataflow controllerexecuting dataflow control functions for the corresponding dataflowexecution graph.
 21. The computer program product in accordance withclaim 20, wherein, for at least one of the plurality of dataflowcontrollers, one of the plurality of role-based dataflow controllerperforms a role of monitoring a health of execution of the dataflowexecution graph assigned to the at least one of the plurality ofdataflow controllers.
 22. The computer program product in accordancewith claim 20, wherein, for at least one of the plurality of dataflowcontrollers, one of the plurality of role-based dataflow controllerperforms a role of checkpointing a state of the dataflow execution graphassigned to the at least one of the plurality of dataflow controllers.23. The computer program product in accordance with claim 20, wherein,for at least one of the plurality of dataflow controllers, one of theplurality of role-based dataflow controller performs a role ofreconfiguring the dataflow execution graph assigned to the at least oneof the plurality of dataflow controllers.
 24. The computer programproduct in accordance with claim 23, the reconfiguring comprisingchanging a structure of the dataflow execution graph.
 25. The computerprogram product in accordance with claim 20, the computer-executableinstructions also being structured such that, when executed by the oneor more processors, the computing system is caused to insatiate at leastone multi-dataflow controller that controls multiple of the plurality ofdataflow controllers.
 26. The computer program product in accordancewith claim 25, the multi-dataflow controller controlling at least someof the multiple of the plurality of dataflow controllers via at leastone intermediate-level dataflow controller.
 27. The method in accordancewith claim 16, the reconfiguring comprising changing a structure of thedataflow execution graph.